The Entropy Gain of Linear Time-Invariant Filters and Some of its Implications

We study the increase in per-sample differential entropy rate of random sequences and processes after being passed through a non minimum-phase (NMP) discrete-time, linear time-invariant (LTI) filter G. For LTI discrete-time filters and random processes, it has long been established that this entropy gain, G(G), equals the integral of log ∣ ∣G(e) ∣ ∣. It is also known that, if the first sample of the impulse response of G has unit-magnitude, then the latter integral equals the sum of the logarithm of the magnitudes of the non-minimum phase zeros of G (i.e., its zeros outside the unit circle), say B(G). These existing results have been derived in the frequency domain as well as in the time domain. In this note, we begin by showing that existing time-domain proofs, which consider finite length-n sequences and then let n tend to infinity, have neglected significant mathematical terms and, therefore, are inaccurate. We discuss some of the implications of this oversight when considering random processes. We then present a rigorous time-domain analysis of the entropy gain of LTI filters for random processes. In particular, we show that the entropy gain between equal-length input and output sequences is upper bounded by B(G) and arises if and only if there exists an output additive disturbance with finite differential entropy (no matter how small) or a random initial state. Unlike what happens with linear maps, the entropy gain in this case depends on the distribution of all the signals involved. Instead, when comparing the input differential entropy to that of the entire (longer) output of G, the entropy gain equals B(G) irrespective of the distributions and without the need for additional exogenous random signals. We illustrate some of the consequences of these results by presenting their implications in three different problems. Specifically: a simple derivation of the rate-distortion function for Gaussian non-stationary sources, conditions for Milan S. Derpich and Matı́as Müller are with the Department of Electronic Engineering, Universidad Técnica Federico Santa Marı́a, Chile. Emails: [email protected],[email protected]. Jan Østergaard is with the Department of Electronic Systems, Aalborg University, Denmark. Email: [email protected]. This work is partially supported by CONICYT PCHA/ Beca Magı́ster Complementario/Año 2013 – Folio 221320226, CONICYT Fondecyt grant 1140384 and CONICYT Basal Fund FB0008. December 14, 2015 DRAFT